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ABSTRACT 1 

To address the challenge of developing critical thinking skills in college students, this empirical 
study examines the effectiveness of cognitive exercises in developing those skills. The study uses 
Critical Thinking: Building the Basics by Walter, Knudsvig, and Smith (2003). This workbook is 
specifically designed to exercise and develop critical thinking skills. The authors use it as the 
intervention tool. Freshman students enrolled in a first-year experience course participated in the 
study; one group used the book, the other did not. Students spent a few weeks in class working 
through the problems, and at the end of the semester, both groups completed the iCritical 
Thinking Skills Test. Analysis of final test scores prove the effectiveness of exercises on improving 
the critical thinking skills of college freshmen. 
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eveloping critical thinking skills is frequently one of the goals found in college and university 
mission statements. Yet evidence suggests students are not learning these skills in school. The 
ability to think critically contributes to successful problem solving and decision making in all areas 
of life, including academic, professional, civic, and even interpersonal relationships. Stephen Brookfield, in an 
interview with James Johanson (2010), said, “Businesses and communities have realized the importance of critical 
thinking skills and are asking colleges to help students build these skills” (p. 30). The majority of instructors 
understand the importance of building these skills, however “numerous barriers to the substantive cultivation of 
critical thinking skills in the classroom continue to exist” (Crenshaw, Hale, & Harper, 2011, p. 13). Since the 1980s, 
ongoing discussion continues about the responsibility of educators to teach students to use these higher level 
thinking skills. Many professionals interested in education and critical thinking recognized this need and attempted 
to address it in their classrooms. However, as Barr & Tagg (1995) suggests, “critical thinking is taught in the same 
way that other courses have traditionally been taught, with an excess of lecture and insufficient time for practice” (p. 
14). Perhaps the insistence on constant lecturing is the reason for dismal test scores reporting college students’ 
critical thinking abilities (Avrum & Roska, 2011). 


LITERATURE REVIEW 


Many definitions of critical thinking are similar in their references to higher-level thinking, problem 
solving, analytical, and questioning approaches to looking for the truth. According to Facione (2011), critical 
thinking is “the process of reasoned judgment” (p. 3). Facione organized The Delphi study in the late 1980s, which 
concluded with a recommendation that becoming skilled in critical thinking requires one to learn how to use these 
abilities effectively in many different contexts, including the roles we take on in society. Scriven and Paul (2011), of 
The Foundation of Critical Thinking, offered this definition: “Critical thinking is that mode of thinking - about any 
subject, content, or problem in which the thinker improves the quality of his or her thinking by skillfully taking 
charge of the structures inherent in thinking and imposing intellectual standards upon them” (para. 10). 


1 Note. From “ Developing Critical Thinking Skills: Assessing The Effectiveness Of Workbook Exercises,” by E.D. Wallace and R.N. Jefferson, 
2013, New Review of Academic Librarianship, 19.3, p. 246-255. Copyright 2013 by Routledge. Reprinted and modified with permission. 
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Bradley Dowden is a professor of philosophy at California State University in Sacramento, and he 
described these skills to his students in this way: “Critical thinking skills involve the ability to reason, to assemble 
evidence in order to develop a position and to communicate complex ideas. . . . improve your skills at making wise 
decisions about what to believe and do” (“What Is the Course About?” para. 1). His course description includes 
examples of how these skills might be used. See Figure 1 for his examples of how these skills might be used. 


Figure 1. Critical thinking skills can be used to: 

• grasp the point that a writer or speaker is trying to make 

• detect whether someone's claim needs more evidence to back it up 

• distinguish between strong arguments and weak ones 

• generate reasons for your viewpoint on some issue 

• decide what information in a piece of writing or speaking to accept and use, and 
decide what information to reject and not use 

• reason from a hypothetical assumption 

• make a potentially strong argument stronger 

• practice conscious quality control as you think 

Reprinted from Dowden (2012) “What Is the Course About” 

The literature has suggested that practice is necessary for building critical thinking skills (Barr & Tagg, 
1995; Facione, 2011; Scriven & Paul, 2011). Mottola and Murphy (2001) concluded from an activity study that 
“critical thinking is most likely to occur and continue when it is supported by others and repeatedly practiced” 
(p. 161). From 2006 to 2009, 19 institutions participated in The Wabash National Study of Liberal Arts Education. 
The study found that “academic experiences that challenged students to analyze ideas and explore their own 
thoughts pushed them to think critically” (Goodman, Baxter Magolda, Seifert, & King, 2011, p. 4). 

Methods thought to be most effective include activities such as group or class discussion, debate, case 
studies, using controversial issues to inspire passion, and motivating students to think more deeply and analytically. 
Unfortunately, these activities usually take time away from the lecture, a traditional method of covering important 
curriculum content. However, at Brigham Young University, the Center for Teaching and Learning found that in- 
class discussions and active learning methods, such as case studies, are just as effective at covering the content. 
Students tend to retain information when actively engaged in working with the subject (Eliason, 2013). 

The earliest stages of critical thinking skills can be developed with practice, using exercises that 
demonstrate a new way of mental processing, developing an awareness of a mode of thinking that is most effective 
in solving problems. These exercises train the mind to operate inquisitively. “Critical thinking skills are cultivated 
through . . . the disciplined use of a question generating language [that] helps students to develop greater awareness 
of course content and self’ (Crenshaw et. al., 2011, p. 21). Brookfield (2006) conducted many studies on the 
pedagogy of critical thinking. He came to believe that the best method of teaching these skills is to model them, 
“seeing a teacher do this in front of their eyes and have them explain by saying, ‘now I’m going to look at my 
assumptions, now I’m going to view this idea that we’ve just learned through different perspectives...’” (Johanson, 
2010, p. 26). The workbook exercises use this type of questioning to model an instructor’s demonstration of the 
critical thinking process. The literature review uncovered few studies using exercises to develop critical thinking 
skills. One study, by Ellen M. Cotter and Carrie Sacco Tally (2009), found that textbook type exercises “did not 
have a positive effect on either formal operational thought or critical thinking” (p. 3). In their study, participants 
completed four written assignments, one per month, over the course of one semester. At the end of this period, 
participants completed the Group Assessment of Logical Thinking (GALT) and California Critical Thinking Skills 
Test (CCTST). However the exercises in the Cotter and Tally study are significantly different from those in the 
present study. 
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METHOD 


Participants 

This study examines the effectiveness of exercises in developing critical thinking skills. A total of 76 
undergraduate students enrolled in a required freshman orientation seminar and participated in the study. All 
participants signed an informed consent form which described and explained the procedures, risks, benefits and 
compensation policy. Only some of the students were given a workbook with cognitive exercises. After all students 
completed a critical thinking exam, the study coordinator collected and compared the data. 

Materials and Procedure 

Only 25 participants received a workbook. Critical Thinking: Building the Basics by Walter, Knudsvig, and 
Smith (2003). Students marked their answers directly on the pages in the book. In the introduction of the book, the 
authors address students directly, explaining how the skills they acquire from using the book will enable them to 
perform at a higher academic level in classes “across the curriculum in any course ... and ensure that [students] have 
every opportunity to become a better learner and thinker” (p. vii). Using the Topic, Class, Description, Relevance 
(TCDR ) strategy, students work through exercises, by answering the following questions found on page five: 

1 . What is the topic I must understand? 

2. What is the overall class to which this topic belongs? 

3. What is the description of the topic? 

4. What is the relevance of the topic? 

These questions guide students to organize the content they are reading. 

In the book, Part I titled “A Basic Strategy for Learning and Thinking Smart,” explains the TCDR strategy 
and includes practice sets to familiarize students with the concepts. Students in the study worked through the 
exercises on page five to 41. The exercises become increasingly more difficult with longer passages and more 
complex content. (See Appendix A - Examples of Critical Thinking Exercises) 

Participants in this study took time in class to complete the exercises. The director of the study prohibited 
outside help or group work. During the first three weeks of the first semester, the instructor scheduled three to four 
class periods for this activity and dedicated the first twenty minutes to the workbook. 

In addition to the 25 students who used the workbook, another 51 students participated in the study without 
a book. At the end of the semester all participants took the iCritical Thinking Skills Test (ETS, 2011). The study 
coordinator documented, charted, and analyzed student test scores to determine whether the workbook exercises 
contributed to improved critical thinking scores. 

The iCritical Thinking exam is supported by Educational Testing Service (ETS), and “features real-time, 
simulated, scenario-based task items designed to measure your students’ ability to navigate, critically evaluate, and 
make sense of the wealth of information available through digital technology” (ETS, p. 2). Student score in seven 
task types: define, access, evaluate, manage, integrate, create, and communicate information. 

Students took the test in the library computer lab. In preparation for the administration of the test, the 
iQsystem test feature needed to be downloaded prior to the test. The library’s technology service coordinator turned 
off each computer’s security software to accommodate the software on a more permanent basis. The security 
software then was turned on again. This was completed prior to testing day. 

The assessment instrument we used was delivered by Certiport, a computer based testing and assessment 
service. Students registered an account online in the iQsystem, for Certiport’s purposes and completed a 
demographic survey. Once logged in, a certified proctor verified the identity of each student, and typed in a 
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protected password. The proctor must be associated with the institution that purchased the test and must view a short 
training tutorial before being certified as a proctor. 

Logistic difficulties are important to point out because although the test period is one hour, the whole 
registration process takes another 15 minutes, making the completion of all 14 tasks challenging. A short tutorial 
shows students how to navigate through the various types of questions. At the conclusion of the test, students had 
the opportunity to print their score report, which shows their score and the required score to pass. 

ETS has renamed the The iCritical Thinking Skills Test to i Skills assessment. 


RESULTS 

Data Analysis 

We used the SPSS Business Analytics software version 17.0 to analyze test results (IBM, 2008). 
Descriptive statistics (minimum, maximum, mean, and standard deviation) of iCritical Exam performance for both 
groups of students is presented Table 1. A large amount of variability exists within each subtest scores. Based on the 
range, the Create and Evaluation subtests have the greatest amount of variability (Range = 100, respectively) with 
Communicate having the least amount of variability (Range = 71). These results differ slightly when compared with 
the standard deviation. The Manage subtest has the greatest amount of variability (SD = 28.3) with Communicate 
still having the least amount of variability (SD = 15.3). 


Table 1. Minimums, Maximums, Means, and Standard Deviations for iCritical Exam Subtest and Total Scores (N=76) 


Subtest/Exam 

Minimum 

Maximum 

Mean(SD) 

Access 

8 

100 

55.4(26.2) 

Communicate 

10 

81 

41.5(15.3) 

Create 

0 

100 

61.9(24.4) 

Define 

17 

93 

60.4(19.4) 

Evaluate 

0 

100 

49.5(27.3) 

Integrate 

0 

94 

49.1(22.3) 

Manage 

0 

100 

41.3(28.3) 

iCritical Exam 

40 

360 

196.7(74.5) 


Students with scores 240 or higher received a passing score on the iCritical Exam. Overall, approximately 
20% (N=76) of the students received a passing score. Table 2 shows the percentage of students with a passing test 
score who used the iCritical workbook (36%) was higher than the percentage of students with passing scores than 
students who did not use the workbook (12%). The Chi-Square test revealed a significant difference between book 
use and performance on the iCritical Exam (X 2 (1) =6.22, p < .05). 


Table 2. Frequency and Percentage of Students Book Use by Pass/Fail Rate 


Exam Performance 

Book Used 

Book Not Used 

Total 

Pass 

9 (60%) 

6 (40%) 

15 (100%) 

Fail 

16 (26%) 

45 (74%) 

61 (100%) 


Table 3 presents the mean and standard deviations for the seven iCritical Exam subtest and total scores by 
book use. Levene’s Test revealed variances to be approximately equal on all subtests. Mean differences were 
statistically significant for the Integrate (f (74 ) = 2.14) and Manage (f {74) = 2.76) iCritical Exam subtests. 
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Table 3. Means and Standard Deviations for iCritical Exam Subtest and Total Scores by Book Use (N=76) 


Subtest/Exam 

Book Used (n,=25) 

Book Not Used (n 2 =51) 

Access 

60.6(24) 

52.9(27.1) 

Communicate 

44.6(18.8) 

40(13.2) 

Create 

68(26) 

59(23.3) 

Define 

61.4(16.8) 

59.8(20.7) 

Evaluate 

53.5(28.5) 

47.5(26.8) 

Integrate 

56.7(22.5) 

43.3(21.4) 

Manage 

53.6(27.7) 

35.3(26.9) 

iCrit Test 

217.6(73.6) 

186.5(73.5) 


A one-way ANOVA was used to examine test performance (pass/fail) for the iCritial Exam subtests. 
Results showed statistically significant differences for two subtests: Integrate (F(i, 74 ) = 4.59, p < .05) and Manage 
(F(l,74) = 7.64, p < .01). The Levene’s Test revealed variances to be approximately equal on all subtests except 
Communication (F(i, 74 )= 4.9, p < .05). 

The amount of time used by students to complete the exam was compared between those who used the 
iCritical workbook and those who did not. Statistically significant results were found between the two groups (t (7 4 , = 
2.22, p < .05). A comparison between students who received a passing score on the exam and those who did not 
revealed a statistically significant difference in the amount of time each group spent taking the test (t< 74 ) = 6.72, p < 
. 001 ). 


DISCUSSION 


Although this research data came from a relatively small sample size, the results reveal that critical thinking 
exercises, such as those in this workbook, can help develop critical thinking skills in college freshmen. As well- 
known author and educator, Clyde Freeman Herreid (2004) said, “by developing habits of mind that speed things 
along,” educators can improve students’ cognitive abilities (p. 64). If future research in this area reaffirms these 
findings, the authors support adopting a tool such as the Critical Thinking: Building the Basics workbook, in a 
freshman orientation course, as effective in developing critical thinking skills. This could be the first systematic step 
to advancing critical thinking skills in post-secondary institutions. 

Limitations of the Study 

Limitations to the study include motivation and length of the test. Students who did not have access to the 
workbook in class did not appear motivated to do well on the test, while those who used the workbook seemed more 
invested in the study and motivated to do their best on the test. The length of the test was a problem because class 
periods at this college are 50 minutes long; however, the time limit for completing the test was sixty minutes. 
Therefore, arranging a time to administer the test became challenging. In addition, time required for instructing and 
registering students to take the test added to this logistical problem. 

Conclusion 

Final data of this study demonstrates the effectiveness of exercises in developing critical thinking skills. 
The exercises were specifically designed to improve thinking skills, and proved successful in doing so. If 
educational institutions adopt similar exercises into the curriculum, our future student/citizens will enter the 
workforce with these much needed skills. 
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Examples of Critical Thinking Exercises 

Practice Item: Read the following definition and complete the matching task below: 


A chair is a piece of furniture consisting of a horizontal surface supported by three or more legs and used 
for sitting on. 

Match the items in column B to the correct label in column A by drawing a line between them. 
Figure 1. TCDR Identification 



B 

used for sitting on 
a chair 

consisting of a flat surface, supported 
by three or more legs 
a piece of furniture 


EXAMPLE: In the following exercises, “one item of each pair is a good answer and tells you what you needed to 
know; the other is bad and gives you incomplete information. 


• Circle each answer as good or bad. 

• Tell which of these pieces of information is missing from the bad answer: topic, class, description, or 
relevance. 


Fish are animals that have backbones and are able to live in water throughout their entire lives. They 
breathe with gills rather than with lungs. In place of legs they have fins. Their bodies are covered with hard 
scales that overlap like roof shingles. They provide humans with one of their most important sources of 
food. 

Question: What are fish? 

good / bad good / bad 

1. a. Fish are animals that live b. Fish are animals that live in 

In water and are used for water and are a source of 

food. food. They have a backbone, 

gills, fins and a covering of scales. 

What part is missing from the bad answer?_ 

good / bad good / bad 

2. a. Fish are animals that live in b. Fish are animals that live in 

water and have a backbone, water. They have a of backbone, 

gills, fins and a covering of gills, fins and a covering of scales, 

scales. They’re used for food. 

What part is missing?_ 
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good / bad 

3. a. Fish live in water. They have 
a backbone, gills, fins and a 
covering of scales. They’re 
used for food. 

What part is missing?_ 
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good / bad 

b. Fish are animals that live in 
water. They have a backbone, 
gills, fins and a covering of 
scales. They’re used for food. 
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